Superintelligence and risks of AI - Lecture 10
According to Goodall, what are the three phases in programming moral machines?
1. Rational (rule-based or outcome-based), 2. AI-based (using training data), and 3. Natural language feedback (XAI). (Page 3)
What are Goodall's critiques of the "Rational" phase approaches?
Rule-based ethics is too minimal for unclassified cases, while outcome-based ethics recommends some morally bad decisions. (Page 3)
What are the problems with the "AI-based" phase (using training data) according to Goodall?
It would reproduce human shortcomings and lack explainability. (Page 3)
According to Awad et al., why is ethics preferred over direct regulation for AI?
Black Box ML makes principles opaque, technology changes faster than regulation, and the source of errors is hard to trace. (Page 5)
What is "SITL" and why do Awad et al. argue for it?
"Society in the loop"; it's considered necessary for a dynamic consensus on AI ethics, which is needed to adopt the benefits of automation. (Page 6)
What general preferences did the Moral Machines project identify?
People prefer sparing the lawful (over unlawful), humans (over pets), the greater number, and younger humans (over older). (Page 7)
What is the superficial definition of risk?
risk = probability $\times$ consequence. (Page 12)
What is the difference between risk, uncertainty, and ignorance?
Risk has a known probability, uncertainty means we cannot attribute a probability, and ignorance means we don't even know the scenarios ('we don't know what we don't know'). (Page 13)
What is the general problem with AI bias found in training data?
The data reflects existing underlying inequalities in society. (Page 16)
What is a "dual-use technology"?
A technology that can be used for both civil and military purposes. (Page 17)
How does AI enhance the threat of disinformation?
AI reduces the cost of generating mis/disinformation at scale. (Page 19)
How is the job displacement risk from AI different this time?
AI targets cognitive tasks (like coding, creativity, and decision-making), not just manual labor. (Page 20)
What is the "enfeeblement and de-skilling" risk described by Stuart Russell?
If everyone stops learning skills that machines already have, the human race will collectively lose its autonomy. (Page 22)
What is the "Singularity"?
A theoretical point where artificial intelligence surpasses human intelligence, becomes uncontrollable, and leads to societal change. (Page 23)
What is "Superintelligence" as defined by Bostrom?
"any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest". (Page 24)
What is the "Orthogonality Thesis"?
The idea that intelligence and final goals are separate (orthogonal), meaning any level of intelligence can be combined with any final goal (e.g., paperclip-maximizing). (Page 25)
What is the "Instrumental Convergence Thesis"?
The idea that intelligent agents with varied final goals will still pursue similar intermediate goals, such as self-preservation and acquiring resources. (Page 26)
What is Chalmers' 3-step argument for the arrival of $AI++$ (superintelligence)?
1. There will soon be AI. 2. If there is AI, there will be $AI+$ (human-level) soon after. 3. If there is $AI+$, there will be $AI++$ soon after. (Page 30)
What is "Value alignment"?
The idea that AI's values will align with human values. (Page 31)
What is "Long-termism"?
The idea that the existence of sentient beings in the distant future should strongly influence our decisions now. (Page 31)
What is an "Existential Risk"?
A risk that threatens the extinction of humanity, civilizational collapse, or the permanent loss of our potential for desirable future development. (Page 32)
What is a "catastrophic risk"?
A non-zero probability of a catastrophic outcome, usually involving harm on a large scale like a pandemic, nuclear war, or climate change. (Page 33)
What is a Type-I error in risk assessment?
A mistake of assuming a statement is true while it is false (a false positive). (Page 35)
What is a Type-II error in risk assessment?
A mistake of assuming a statement is false while it is true (a false negative). (Page 35)
What are four key considerations in the "ethics of risk" (i.e., when is it permissible to impose risks)?
Do benefits outweigh costs? Are risks fairly distributed? Would people agree to the risks? What are the alternatives? (Page 37)
According to McKinsey (2023), which job sectors will see the largest *decrease* in labor demand by 2030 due to AI?
Office support (–18%) and Customer service and sales (–13%). (Page 38)
According to McKinsey (2023), which job sectors will see the largest *increase* in labor demand?
Health professionals (+30%), Health aides/technicians (+30%), and STEM professionals (+23%). (Page 38)
What is "Prioritarianism"?
A principle of justice that gives priority to the worse off over the better off. (Page 39)
What are the four risk categories in the EU AI Act?
Unacceptable Risk, High Risk, Limited Risk, and Minimal Risk. (Page 40)
What is "deep uncertainty"?
A situation where it is impossible to predict a full range of realistic possible outcomes. (Page 43)
What is the "Precautionary Principle" (from the Rio Declaration)?
Where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures. (Page 42, 44)
What did the "Pause Giant AI Experiments" open letter call for?
A pause of at least 6 months on the training of AI systems more powerful than GPT-4. (Page 45)